A payout job failed silently for nine days

tally.app moves money on Friday mornings. Thirty-one thousand contractors, most of them paid within an hour of their client approving the invoice, and none of them inclined to be patient about it.

In March a retry in the payout worker started swallowing its own exception. One batch in forty never left the queue. Nothing in the dashboard changed, nothing in the logs stood out, and the first anyone heard of it was a support ticket nine days later from a drywall contractor in Nampa who had been paid twice for the same job and not at all for the next one.

What the nine days cost

Two engineers spent a week reading queue logs by hand. The finance team re-ran 1,900 payouts and reconciled them against the ledger line by line. Support answered 214 tickets that all said the same thing in different words.

• Eleven days of engineering time, counting the reconciliation.
• $4,800 in wire fees for the manual re-runs.
• Two customers on annual contracts who did not renew, and said why.

The part that actually changed

Artisan went in on a Tuesday afternoon. The SDK took one package and a DSN; queued jobs report as their own transactions, so a worker that throws inside a retry is still an event with a stack, a release and the commit that last touched the file.

The same week, a different retry started failing. The alert reached the on-call channel in four minutes, pointed at app/Payouts/Dispatch.php:141, and named the deploy from that morning as the suspect. The fix shipped before lunch.

We did not get better at reading logs. We stopped having to.

A year in

tally.app sends about 1,284 events a day now, which group into eleven open issues — a number the team can actually work through on a Monday. Repeat incidents are down 71% year over year, and the payout worker has its own project, its own alert rule and its own budget for how slow it is allowed to get before somebody is paged.